Skip to content

feat(dogfood): metadata-driven verifier (objectstack verify) — auto-derive runtime proofs#2027

Merged
os-zhuang merged 1 commit into
mainfrom
feat/dogfood-auto-verify
Jun 18, 2026
Merged

feat(dogfood): metadata-driven verifier (objectstack verify) — auto-derive runtime proofs#2027
os-zhuang merged 1 commit into
mainfrom
feat/dogfood-auto-verify

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Why

The dogfood gate (#2020, #2022) pins the framework's own example apps. But the platform's value is that an AI authors arbitrary metadata for a third-party app — and framework CI never sees that app's metadata. A bug that only manifests with their objects ships green.

This is the seed of objectstack verify: derive a runtime contract from an app's metadata alone (no hand-written tests), then run it against the real in-process stack. The consumer (e.g. hotcrm) runs it on their own app.

What

v0 derives per-object CRUD round-trip: synthesize a valid create body from each object's field types → POST → GET → assert type fidelity.

  • src/derive.ts — metadata → CRUD cases; reports objects blocked by an un-synthesizable required field (e.g. required lookup) instead of silently skipping.
  • src/verify.ts — runs over real HTTP; classifies verified / fidelity-gaps (type leaks) / needs-fixture (auto-record tripped the app's own validation rules) / create-failed·read-failed (real failures) / skipped.
  • harness.ts — wire LocalCryptoProvider (as objectstack dev does) so Field.secret apps are exercisable.
  • test/auto-verify.dogfood.test.ts — runs on CRM + showcase; gates on "no object fails to create/read". Auto-found f_progress's fidelity gap that the hand-written field-zoo matrix missed.
  • test/verify-external.dogfood.test.tsOS_VERIFY_ARTIFACT=<app>/dist/objectstack.json points it at any app's built artifact (plain JSON, no cross-repo hazard); skipIf-gated, never runs in CI.

Proven against hotcrm (the real third-party case)

hotcrm — separate repo, 15 objects, zero dogfood dependency, whose entire runtime verification today is a static smoke test. Pointed at its artifact, the verifier booted it in-process, seeded 162 records, auto-verified:

  ✓ 5 objects verified (account/campaign/forecast/knowledge_article/task)
  ⚠ crm_case   customer_rating <rating>: wrote 3 → read "3"
  ⚠ crm_lead   rating <rating>:          wrote 3 → read "3"
  ⚠ crm_product sku <text>:              wrote "verify-sample" → read "VERIFY-SAMPLE"
  – 7 skipped (required lookup/master-detail need targets)
  • rating → the platform type-leak, proven to propagate into a real consumer app.
  • crm_product.sku → a hotcrm-specific write divergence (uppercased) that no framework test could reach — only the verifier run on hotcrm's metadata surfaces it. (May be intended; the verifier reports a runtime contract divergence, not a pass/fail oracle.)

Notes

  • Private package — no changeset. Full dogfood suite: 27 passed | 3 expected-fail | 1 skipped.
  • Directly answers "does dogfood solve the third-party problem?": the framework-side gate is blind to a consumer's metadata; the shipped metadata-driven verifier run on their app closes it.
  • Roadmap: RLS cross-owner derivation (fix(security)[P0]: enforce RLS on by-id writes — close member-edits-others'-records hole (#1985) #1994 invariant — needs multi-user harness + sharing service); richer fixtures; a real objectstack verify CLI. Scope per ADR-0054 (#2024).

🤖 Generated with Claude Code

…erive runtime proofs

The dogfood gate pins the framework's own example apps; it never sees a third
party's metadata. But the platform's value is that an AI authors arbitrary
metadata and it works at runtime — for a SPECIFIC consumer app the framework CI
can't reach.

This adds the seed of `objectstack verify`: derive a runtime contract from an
app's metadata ALONE (no hand-written tests), then run it against the real
in-process stack. v0 derives per-object CRUD round-trip — synthesize a valid
create body from each object's field types, POST it, GET it back, assert type
fidelity.

- src/derive.ts — metadata → CRUD cases. Synthesizes values per field type;
  reports objects blocked by un-synthesizable required fields (e.g. a required
  lookup) rather than silently skipping.
- src/verify.ts — runs the derived cases over real HTTP, returns a structured
  report. Classifies outcomes: verified / fidelity-gaps (type leaks) /
  needs-fixture (the auto-record tripped the app's own validation rules) /
  create-failed|read-failed (real runtime failures) / skipped.
- harness.ts — wire LocalCryptoProvider (as `objectstack dev` does) so apps with
  a Field.secret are exercisable end-to-end.
- test/auto-verify.dogfood.test.ts — runs it on CRM + showcase; gates on "no
  object fails to create/read". On showcase it auto-found f_progress's
  type-fidelity gap too — which the hand-written field-zoo matrix had missed.
- test/verify-external.dogfood.test.ts — OS_VERIFY_ARTIFACT=<app>/dist/objectstack.json
  points it at ANY app's built artifact (the consumer entry point; skipped in CI).

Proven against hotcrm (15-object third-party app, separate repo, zero dogfood
dependency): booted in-process, seeded 162 records, auto-verified — found the
platform rating type-leak in crm_case/crm_lead AND a hotcrm-specific write
divergence (crm_product.sku normalized to uppercase) that no framework test
could reach.

Private package — no changeset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
spec Ready Ready Preview, Comment Jun 18, 2026 6:48am

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

No hand-written docs reference the 1 changed package(s). ✅

@os-zhuang
os-zhuang merged commit 311bcc6 into main Jun 18, 2026
15 checks passed
@os-zhuang
os-zhuang deleted the feat/dogfood-auto-verify branch June 18, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant